Skip to content

[9.3] (backport #15427) Override fleet.ssl.certificate_authorities from env vars in container mode#15467

Open
mergify[bot] wants to merge 1 commit into
9.3from
mergify/bp/9.3/pr-15427
Open

[9.3] (backport #15427) Override fleet.ssl.certificate_authorities from env vars in container mode#15467
mergify[bot] wants to merge 1 commit into
9.3from
mergify/bp/9.3/pr-15427

Conversation

@mergify

@mergify mergify Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

When in container mode, override fleet.ssl.certificate_authorities from env vars FLEET_CA, KIBANA_CA, and ELASTICSEARCH_CA instead of using saved fleet.enc values.

Follows the same first-set-wins fallback chain used in setup_config.go.

Why is it important?

Without this override, an agent running in container mode will crash during startup if fleet.enc still contains a fleet.ssl.certificate_authorities path that no longer exists on disk.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Disruptive User Impact

N/A

How to test this PR locally

Unit test: TestContainerEnvOverridesFleetTLSPaths
Integration test: TestContainerCMDTLSCertOverride
Manually: Using kind, deploy ECK cluster. Enroll an agent with FLEET_SERVER_CLIENT_AUTH=required and FLEET_CA pointing to a mounted CA bundle. Unmount the CA bundle, unset env vars, and verify that agent restarts healthy.

Related issues

Closes #15408
Relates #14408

This is an automatic backport of pull request #15427 done by Mergify.

@lorienhu @mergify
Override fleet.ssl.certificate_authorities from env vars in container…
4bb7ecd 
… mode (#15427)

Reads fleet.ssl.certificate_authorities from the first-set of FLEET_CA, KIBANA_CA, ELASTICSEARCH_CA over potentially stale fleet.enc values.

(cherry picked from commit 3ca03cf)
@mergify mergify Bot added the backport label Jul 11, 2026
@mergify mergify Bot requested a review from a team as a code owner July 11, 2026 12:46
@mergify mergify Bot requested review from macdewee and removed request for a team July 11, 2026 12:46
@mergify mergify Bot added the backport label Jul 11, 2026
@mergify mergify Bot requested a review from samuelvl July 11, 2026 12:46
@mergify mergify Bot mentioned this pull request Jul 11, 2026
Merged
6 tasks
@github-actions github-actions Bot added the bug Something isn't working label Jul 11, 2026
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 11, 2026

Copy link
Copy Markdown

⏳ Build in-progress, with failures

Failed CI Steps

cc @lorienhu

@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

TL;DR

The FIPS-only unit job fails deterministically in the new TestContainerEnvOverridesFleetCA regression test, before it exercises the CA override. The test calls store.Save to create fleet.enc, but the 9.3 backport still uses cipher.NewGCM in internal/pkg/agent/vault/aesgcm/aesgcm.go; Go FIPS 140-only rejects that arbitrary-IV GCM constructor.

Remediation

  • Add fipsutils.SkipIfFIPSOnly(t, "the 9.3 vault implementation uses cipher.NewGCM, which is unavailable in FIPS 140-only mode") at the start of TestContainerEnvOverridesFleetCA (or otherwise avoid encrypted-store setup in this test).
  • Keep the test enabled in non-FIPS unit runs; alternatively, backport the complete vault AES-GCM random-nonce migration if that broader change is intended for 9.3.
  • Re-run GODEBUG="fips140=only,tlsmlkem=0" .buildkite/scripts/steps/unit-tests.sh and the targeted internal/pkg/agent/cmd tests.
Investigation details

Root Cause

At internal/pkg/agent/cmd/container_test.go:1458-1461 in the failing revision, TestContainerEnvOverridesFleetCA invokes store.Save, which encrypts the generated fleet.enc. The backport's internal/pkg/agent/vault/vault_file_notwindows.go:28-37 delegates encryption to aesgcm.Encrypt, and internal/pkg/agent/vault/aesgcm/aesgcm.go:54 calls cipher.NewGCM(block). Under GODEBUG=fips140=only, that call returns the exact error reported by the build. The existing vault/storage tests are already skipped in this mode for the same incompatibility; this new test lacks that guard.

Evidence

  • Build: https://buildkite.com/elastic/elastic-agent/builds/42812
  • Job/step: Unit tests - fips140=only Ubuntu 22.04
  • Key log excerpt: vault Set: could not encrypt key: crypto/cipher: use of GCM with arbitrary IVs is not allowed in FIPS 140-only mode, use NewGCMWithRandomNonce
  • The failure is reported at internal/pkg/agent/cmd/container_test.go:1315 while the test is saving encrypted state, not while loading or asserting CA configuration.

Verification

Static analysis of the failing log, PR diff, and commit 4bb7ecd; no local FIPS test was run.

Follow-up

No existing flaky-test issue or prior PR review thread matched this root cause.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@macdewee macdewee Awaiting requested review from macdewee macdewee is a code owner automatically assigned from elastic/elastic-agent-control-plane
@samuelvl samuelvl Awaiting requested review from samuelvl samuelvl is a code owner automatically assigned from elastic/elastic-agent-control-plane

At least 1 approving review is required to merge this pull request.

Assignees

@lorienhu lorienhu

Labels

backport bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lorienhu